一、diffusers版本0.32.0时模型量化问题1、bitsandbytes运行正常但是采样速度变慢需要下载对应分支pip install githttps://github.com/BenjaminBossan/peftfix-low-cpu-mem-usage-bnb-8bittransformer_8bit FluxTransformer2DModel.from_pretrained(black-forest-labs/FLUX.1-dev,subfoldertransformer,quantization_configDiffusersBitsAndBytesConfig(load_in_8bitTrue),torch_dtypetorch.bfloat16,)pipe FluxPipeline.from_pretrained(black-forest-labs/FLUX.1-dev,transformertransformer_8bit,torch_dtypetorch.bfloat16,).to(cuda)pipe.load_lora_weights(hf_hub_download(ByteDance/Hyper-SD, Hyper-FLUX.1-dev-8steps-lora.safetensors),adapter_namehyper-sd)pipe.set_adapters(hyper-sd, adapter_weights0.125)2、optimum导入lora时存在问题待解决Traceback (most recent call last):File /home/iv/anaconda3/lib/python3.10/multiprocessing/process.py, line 314, in _bootstrapself.run()File /home/iv/anaconda3/lib/python3.10/multiprocessing/process.py, line 108, in runself._target(*self._args, **self._kwargs)File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/app.py, line 112, in process_dataIVmodelHandle.loadModel()File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/BaseModel.py, line 28, in loadModelself.model GlobalConfig.get_model()File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/GlobalConfig.py, line 76, in get_modelmodelHandler model()File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/FluxTxt2Img/Flux_txt2img.py, line 145, in __init__self.pipe.load_lora_weights(hf_hub_download(Shakker-Labs/FLUX.1-dev-LoRA-Logo-Design, FLUX-dev-lora-Logo-Design.safetensors), adapter_namelogo)File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/66v/lib/python3.10/site-packages/diffusers/loaders/lora_pipeline.py, line 1559, in load_lora_weightstransformer_lora_state_dict self._maybe_expand_lora_state_dict(File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/66v/lib/python3.10/site-packages/diffusers/loaders/lora_pipeline.py, line 2085, in _maybe_expand_lora_state_dictbase_weight_param transformer_state_dict[base_param_name]KeyError: single_transformer_blocks.0.attn.to_k.weight._data二、某些lora加载失败解决办法diffusers版本过低升级diffusersTraceback (most recent call last):File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP-test/algos/AnystyleImg2Img/newfilter.py, line 1042, in modulemodel AnyStyleImg2ImgModel()File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP-test/algos/AnystyleImg2Img/newfilter.py, line 725, in __init__self.model.pipline.pipe.load_lora_weights(load_file(os.path.join(self.model_root, self.config[SDXL][self.default_style][lora_repo][0]), deviceself.device), adapter_namemix_lora)File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/venv310/lib/python3.10/site-packages/diffusers/loaders/lora.py, line 1255, in load_lora_weightsself.load_lora_into_text_encoder(File /home/iv/Algo_new/LouHaijie/IVAlgoHTTP/venv310/lib/python3.10/site-packages/diffusers/loaders/lora.py, line 555, in load_lora_into_text_encoderrank[rank_key] text_encoder_lora_state_dict[rank_key].shape[1]KeyError: text_model.encoder.layers.0.self_attn.out_proj.lora_B.weight三、如何在diffusers0.31.0版本使用flux-redux1、安装diffuser0.31.0版本基础模型的pipeline的类和方法都从该库中调用2、从github上安装diffusers0.32.0-release版本flux-redux的类的方法都从该库中调用3、修改diffusers0.32.0-release库diffusers/pipelines/pipeline_utils.pyfrom diffusers import 修改为from [当前库名] import4、修改diffusers0.31.0库diffusers/models/attention_processor.py:hidden_states F.scaled_dot_product_attention(query, key, value, dropout_p0.0, is_causalFalse)修改为hidden_states F.scaled_dot_product_attention(query, key, value, attn_maskattention_mask, dropout_p0.0, is_causalFalse)四、diffusers版本为0.31.0时使用optimum量化输出图片为噪声修改optimum/quanto/tensor/weights/qbytes.py.and torch.cuda.get_device_capability(data.device)[0] 8为and torch.cuda.get_device_capability(data.device)[0] 20修改后if (qtype qtypes[qfloat8_e4m3fn]and activation_qtype is Noneand scale.dtype in [torch.float16, torch.bfloat16]and len(size) 2and (data.device.type cuda and torch.version.cuda)and axis 0and torch.cuda.get_device_capability(data.device)[0] 8):out_features, in_features sizeif (in_features 64and out_features 64and ((in_features % 64 0 and out_features % 128 0)or (in_features % 128 0 and out_features % 64 0))):return MarlinF8QBytesTensor(qtype, axis, size, stride, data, scale, requires_grad)五、diffusers要同时设两个lora权重时有一个不生效果可能是设权重的方式错了错误方式self.pipe.set_adapters([LoraA], adapter_weights[A])self.pipe.set_adapters([LoraB], adapter_weights[B])正确方式self.pipe.set_adapters([LoraA, LoraB], adapter_weights[A, B])
